Field of Art
The disclosure generally relates to task execution and specifically to the dynamic construction and management of task pipelines.
Description of the Related Art
A server environment performs two types of services, front-end (user facing) services and back-end (non-user facing) services. Back-end services include asynchronous and/or short-lived processing jobs, such as collecting and indexing data from remote systems or processing requests by the user that take longer than a few seconds. Typically, an execution infrastructure within the server environment manages the execution of tasks associated with these back-end services.
The execution infrastructures that exist today have several limitations. These infrastructures do not orchestrate tasks that require the coordination of multiple different types of tasks, do not seamlessly handle retries for failed tasks and do not provide the means to resolve race conditions between different tasks. Further, in some cases, the back-up functionality provided by the execution infrastructure is not robust to system failures.
Accordingly, there is a need for a system that enables the execution of multiple tasks of a pipeline in a robust manner.